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(U) Results in Brief 


(U) Evaluation of Contract Monitoring and Management 


for Project Maven 


(U) The objective of this evaluation was to 
determine whether the Army Contracting 
Command (ACC), the Army Research 
Laboratory (ARL), and the Algorithmic 
Warfare Cross-Functional Team (AWCFT, 
also known as Project Maven) monitored 
Project Maven contracts in accordance with 
the Federal Acquisition Regulation (FAR), 
Defense Federal Acquisition Regulation 
Supplement (DFARS), and DoD policy. 
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{EU To support AI development, the 
AWCFT partnered with the ACC to award 
four contracts and one cooperative 
agreement. Federal procurement guidelines 
define the structures, mechanisms, and 
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{E9H roles for conducting contract administration, 
surveillance, and monitoring to track contract performance 
and contract deliverables and to mitigate risks. The primary 
roles that support contract monitoring and management 
include the Contracting Officer (KO), the Contracting Officer 
Representative (COR), and the Technical Monitor (TM). 


(U) The ACC-Aberdeen Proving Ground Contracting Center KO 
appointed a Contracting Officer Representative (COR) for 
Project Maven’s four contracts and one cooperative agreement 
in accordance with the FAR, and the Army Research 
Laboratory COR and the AWCFT TM successfully monitored 
and managed Project Maven’s four contracts and one 
cooperative agreement in accordance with the FAR, DFARS, 
Defense Grant and Agreement Regulatory System, and 
contract requirements. Specifically, the TM, with the support 
of the AWCFT, actively monitored contract deliverables 

using AWCFT-developed reporting, metrics, processes, 

and procedures to monitor and manage the Project Maven 
contracts and meet Project Maven objectives. 


(U) However, the AWCFT did not document its approach 
to monitoring by formalizing the reporting metrics, 
processes, and procedures for monitoring and managing 
Project Maven contracts. 


(U) Without formalized and documented processes, there is 
an increased risk of lapses occurring in the monitoring and 
management of the Project Maven contracts as the program 
grows and as project personnel change. This could negatively 
affect the long-term success and growth of the project. 


(U) In addition, as AI and machine learning programs expand 
within the DoD, future DoD acquisitions related to this 
complex, rapidly moving technology may not benefit from the 
monitoring and management lessons learned. 
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(U) Results in Brief 


(U) Evaluation of Contract Monitoring and Management 


for Project Maven 


(U) Findings (cont’d) 


(U) Best practices that can further refine and develop 
these types of acquisitions, as well as help provide 
continuity for management and monitoring of emergent 
technology, could be lost if these processes are 

not documented. 


(U) Recommendations 


(U) We recommend that the Chief of the AWCFT 
formalize Project Maven’s processes and procedures 

for monitoring and managing AI development contracts 
to ensure knowledge management, continuity, and 
efficiency when the project is transferred to a mission 
owner and for reference by subsequent novel technology 
projects and cross-functional teams. 


(U) We recommend that the Assistant Secretary 

of Defense for Acquisition conduct a review of the 
AWFCT’s emerging technology acquisition processes 
and procedures to evaluate whether any of the 
processes and procedures used by the AWCFT should be 
further formalized in acquisition policies, implemented 
by other programs and cross-functional teams, or 
incorporated into relevant curriculum and training. 


(U) Management Comments 


and Our Response 


f€98H Throughout the evaluation, we discussed the 
need to formalize Project Maven’s processes and 
procedures with AWCFT personnel and AWCFT 
regularly shared draft documentation in support of 
that need. On October 14, 2021, we discussed the 
specific recommendations of this report with the 
AWCFT. During the discussion, the AWCFT Chief 
stated that the Joint Requirements Oversight Council 
Memorandum supporting the transition of Maven’s | 
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(U) Additionally, following the October 14, 2021, 
discussion, the AWCFT provided evidence of actions 
taken to address the recommendation in this report, 
including a description of roles and responsibilities, 
standard operating procedure, and a Maven 
acquisition guide. The AWCFT also provided us the 
Joint Requirements Oversight Council Memorandum 
supporting the transition of Maven to a mission owner, 
which also outlined several requirements to ensure 
continuity and a seamless transition. These actions 
meet the intent of the recommendation; therefore, 
the recommendation is closed. 


(U) The Assistant Secretary of Defense for Acquisition 
concurred with the recommendation to review the 
AWFCT’s emerging technology acquisition process 

for possible inclusion in DoD acquisition guidance. 

To support this effort, the Office of the Assistant 
Secretary of Defense for Acquisition will interact with 
the Project Maven team to gather additional details, 
such as approaches that dynamically handle changing 
performance metrics and needs, and determine if 

and how those approaches can be elaborated upon 

in DoD acquisition guidance. The comments meet 

the intent of the recommendation; therefore, the 
recommendation is resolved, but will remain open. 
We will close the recommendation when the Assistant 
Secretary of Defense for Acquisition provides us the 
results of the review of Project Maven’s practices and 
resulting determination of whether those practices were 
incorporated into DoD acquisition guidance. 


(U) Please see the Recommendations Table on the next 
page for the status of the recommendations. 


(U) Recommendations Table 


(U) Recommendations | (U) Recommendations | (U) Recommendations 


(U) Management 


Unresolved Resolved Closed 
(U) Assistant Secretary of 
Defense for Acquisition Mone e Mane 
(U) Chief, Algorithmic Warfare Nene Nene 1 
Cross-Functional Team 


(U) Please provide Management Comments by February 7, 2022. 


(U) Note: The following categories are used to describe agency management’s comments to 
individual recommendations. 


e (U) Unresolved — Management has not agreed to implement the recommendation or has not proposed actions 
that will address the recommendation. 


e (U) Resolved — Management agreed to implement the recommendation or has proposed actions that will address 
the underlying finding that generated the recommendation. 


e (U) Closed — DoD OIG verified that the agreed upon corrective actions were implemented. 
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INSPECTOR GENERAL 

DEPARTMENT OF DEFENSE 

4800 MARK CENTER DRIVE 
ALEXANDRIA, VIRGINIA 22350-1500 


January 6, 2022 


(U) MEMORANDUM FOR ASSISTANT SECRETARY OF DEFENSE FOR ACQUISITION 
CHIEF, ALGORITHMIC WARFARE CROSS-FUNCTIONAL TEAM 


(U) SUBJECT: Evaluation of Contract Monitoring and Management for Project Maven 
(Report No. DODIG-2022-049) 


(U) This final report provides the results of the DoD Office of Inspector General’s evaluation. 
We previously provided copies of the draft report and requested written comments on 

the recommendations. We considered management’s comments on the draft report when 
preparing the final report. These comments are included in the report. 


(U) This report contains one recommendation that is considered resolved, but remains open. 
As described in the Recommendations, Management Comments, and Our Response section 
of this report, we will consider the recommendation closed when the Assistant Secretary 

of Defense for Acquisition provides us the results of the review of Project Maven’s practices 
and resulting determination of whether those practices should be incorporated into DoD 
acquisition guidance. We will track this recommendation until an agreement is reached 

on the actions that you will take to address the recommendation and you have submitted 
adequate documentation showing that all agreed-upon actions are completed. 


(U) DoD Instruction 7650.03 requires that recommendations be resolved promptly. Therefore, 
please provide us within 30 days your response concerning specific actions in process or 
alternative corrective actions proposed on the recommendations. Send your response to 


(U) If you have any questions or would like to meet to discuss the evaluation, please contact 
We appreciate the cooperation and assistance received 
during the evaluation. 


Dana ~\ Sehnaon 


Dana Johnson 

Acting Assistant Inspector General for 
Programs, Combatant Commands, and 
Contingency Operations 
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(U) Introduction 
(U) Objective 


(U) The objective of this evaluation was to determine whether the Army Contracting 
Command (ACC), the Army Research Laboratory (ARL), and the Algorithmic Warfare 
Cross-Functional Team (AWCFT), also known as Project Maven) monitored Project 
Maven contracts in accordance with the Federal Acquisition Regulation (FAR), 
Defense Federal Acquisition Regulation Supplement (DFARS), and DoD policy. 


(U) Background 


(U) According to the Brookings Institute, AI and machine learning are emerging, 
fast-moving technologies.? AI refers to software that performs a task or series of 
tasks that normally require human intelligence to perform. Machine learning is a 
subcategory of AI in which the system uses algorithm-based learning to process data, 
improving its functioning as it receives more data. In Al and machine learning, data 
sets are used to train algorithms for rapid development, fielding, research, testing, 


and evaluation. 


f€UH When the AWCFT was established in 2017, its mission was to field technology 
to augment or automate processing, exploitation, and dissemination of full-motion 


video for intelligence support. 


3 (U) The Brookings Institution, “How Artificial Intelligence is Transforming the World,” April 24, 2018. 


4 (U) Deputy Secretary of Defense Memorandum, “Establishment of Algorithmic Warfare Cross-Functional Team 
(Project Maven),” April 26, 2017. 


Introduction 


DODIG-2022-049 | 1 


Introduction 


2 | DODIG-2022-049 


CUI 


(U) The AWCFT Is a Secretary of Defense-Empowered 
Cross-Functional Team 

(U) According to a January 2020 Government Accountability Office report, 
cross-functional teams rely on individuals with different types of expertise to 
work toward a common, well-defined goal and are thought to deliver better and 
faster solutions to complex and fast-moving problems.° 


(U) Congress Directed the DoD to Use Cross-Functional Teams 


(U) To address organizational challenges within the DoD, Congress directed the 
Secretary of Defense to issue an organizational strategy that identified critical 
objectives that would benefit from the use of cross-functional teams. In the 
National Defense Authorization Act of 2017, section 911, Congress stated that one 
purpose of cross-functional teams is to “provide for effective collaboration and 
integration across organizational and functional boundaries in the Department of 
Defense.”® Additionally, section 911 directed the DoD to issue guidance on how to 
set up and operate cross-functional teams and provide team members and civilian 
leaders in the Office of the Secretary of Defense with training in elements of 
successful cross-functional teams. 


5 (U) Government Accountability Office, GAO-20-312, “Defense Management More Progress Needed for 
DoD to Meet Outstanding Statutory Requirements to Improve Collaboration,” January 2020. 


6 (U) Public Law 114-328, “National Defense Authorization Act of 2017,” section 911, December 23, 2016. 


Cut 


cur Introduction 


(U) Cross-Functional Teams Emphasize Speed 


(U) Project Maven Contracts 

(U) To support the objectives and mission of Project Maven, the Deputy Secretary 
of Defense, in DTM 18-002, directed the AWCFT to contract with commercial 
industry. The AWCFT partnered with the Army Research Laboratory (ARL) to 
solicit proposals to address Project Maven objectives. Between September 2018 
and February 2020, the AWCFT and ACC awarded four contracts and 

one cooperative agreement to support Project Maven.® Each of these awards 
addressed a problem set related to Project Maven’s objectives. 


e (U) Three contracts were awarded to ECS Federal, LLC: 


| (85 PAVEMENT arr 
st 


7 (U) “Deputy Secretary of Defense Memorandum,” April 26, 2017. 
a a 


> (cu 
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e (U) One contract was awarded to MORSE Corporation: 


<< 


e (U) One cooperative agreement was awarded to Carnegie Mellon 
University (CMU) and its team of academic institutions: 


<7 


(U) Monitoring Contract Performance 

(U) Federal acquisition regulations provide the standard processes and guidelines 
for the procurement of goods and services by the Executive branch of the 
Government, including the DoD. The FAR is the primary regulation used by 
executive agencies in their acquisition activities. To provide uniform acquisition 
policies and procedures for the DoD, the Office of the Assistant Secretary of 
Defense for Acquisition issues the DFARS and the DFARS Procedures, Guidance, 
and Information regulations, which are the primary regulations that govern DoD 
procurement activities. In addition to the FAR and DFARS, the Defense Grant and 
Agreement Regulatory System (DGARS) is the collection of policies and procedures 
that establish rules specifically for DoD grants and cooperative agreements. 


(U) The FAR, DFARS, and DGARS also define the structures, mechanisms, and roles 
for conducting contract administration, surveilling and monitoring to track contract 
performance, accepting contract deliverables, and mitigating risks. In addition to the 
terms and conditions outlined in contracts, the performance work statement (PWS) 
and the quality assurance surveillance plan (QASP) provide a framework for 
monitoring contractor performance and surveillance of contract deliverables. 


(U) CONTRACT ADMINISTRATION 

(U) The FAR outlines the contract administration activities related to the 
monitoring of suppliers, contractors, and service providers’ performance to fulfill 
the contract obligations.’? Contract administration also ensures appropriate 
actions are taken to remedy any deficiencies observed in contract implementation, 
scope, or terms and conditions. The contract administration ends with the final 
inspection and acceptance of the goods. 


10 (U) FAR Part 42, “Contract Administration and Audit Services,” Subpart 42.3, “Contract Administration Office Functions.” 


Cut 


(U) PERFORMANCE WORK STATEMENTS 

(U) The FAR outlines the elements of a PWS, which describes the required results 
of the contract, establishes measureable performance standards, and relies on 
those standards and financial incentives in a competitive environment to encourage 
competitors to develop and institute innovative and cost-effective methods of 
performing the work.!! The PWS is incorporated into the final contract award and 
is adopted at the start of a contract. 


(U) QUALITY ASSURANCE SURVEILLANCE PLANS 

(U) According to the ACC QASP template, the purpose of the QASP is to provide the 
Government and contractor with evaluation criteria that determines whether the 
performance standards for a specific contract have been met.!2 The QASP outlines 
all work that requires surveillance and the method of surveillance and includes 

the performance standards and acceptable quality level for acceptance of contract 
deliverables. The Contracting Officer (KO) receives the results of quality assurance 
inspections and may use them to document contractor performance. QASPs are 
prepared in conjunction with the preparation of the PWS and are tailored to the 
work effort addressed by the contract. 


(U) Roles and Responsibilities for Project Maven 

Contract Management 

(U) The primary roles that support contract administration and monitoring 
include the KO; the Contracting Officer Representative (COR) or, in the case 

of the cooperative agreement, a Cooperative Agreement Manager; and the 
Technical Monitor (TM). For the four Project Maven contracts and one cooperative 
agreement, personnel from several DoD entities fulfilled these roles. 


(U) Army Contracting Command: Contracting Officer 

(U) The Army Contracting Command (ACC) is a subordinate command of U.S. Army 
Material Command. The Army Contracting Command-Aberdeen Proving Ground 
Contracting Center is responsible for acquiring quality, technologically superior, 
next-generation equipment and services in the shortest time while obtaining best 
value. The Army Contracting Command-Aberdeen Proving Ground Contracting 
Center is a full service, life-cycle acquisition organization. 


(U) Army Contracting Command-Aberdeen Proving Ground Contracting Center 
personnel serve as KOs for the four Project Maven contracts and one cooperative 
agreement. The KO is primarily responsible for managing the high-level 


11 (U) FAR Part 37, “Service Contracting,” Subpart 37.602, “Performance Work Statement.” 


12 (U) The QASP template can be found on the Defense Pricing and Contracting, Contingency Contracting, Defense 
Contingency COR Handbook website, https://www.acq.osd.mil/dpap/ccap/cc/corhb/HTML/documents.html. 


Cut 


Introduction 
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(U) administration of the contract. The duties of a KO include reviewing post- 
award contract information to validate the specified terms and conditions in the 
contract; appointing a COR; appointing a TM, if required; reviewing and approving 
the QASP to ensure it supports quality assurance, acceptance, and delivery of 
contract items; and reporting and documenting any defects or performance issues 
with contract technical and quality requirements. 


(U) Army Research Laboratory: Contracting Officer 
Representative and Cooperative Agreement Manager 

(U) The U.S. Army Combat Capabilities Development Command Army Research 
Laboratory (ARL) is the Army’s national research laboratory and is part of the 
Army Futures Command. The ARL focuses on cutting-edge scientific discovery, 
technological innovation, and transition of knowledge from commercial partners 
to DoD personnel. 


(U) The Associate Branch Chief of the ARL Technology Integration Branch was 
appointed by the KO and serves as both the COR and the Cooperative Agreement 
Manager for the four Project Maven contracts and cooperative agreement. 

The COR’s primary responsibility is managing the day-to-day operations and 
monitoring progress of the contract by directly engaging with the contractor. 
The COR is appointed by the KO and his or her specific duties, usually relating to 
contract administration and technical monitoring, are outlined in the contract. 
The COR must be a Federal employee, be appointed in writing, complete required 
training, and, in some cases, meet specific experience requirements, if the contract 
requires specialized knowledge. The CMU award is a cooperative agreement, 
rather than a contract, and the COR-equivalent role is the Cooperative Agreement 
Manager. The day-to-day responsibilities of the Cooperative Agreement Manager 
are similar to those of a COR. 


(U) Algorithmic Warfare Cross-Functional Team: 

Technical Monitor 

(U) The KO may appoint a TM to assist in monitoring contractor performance if the 
COR requires additional support. The TM, similar to a COR, is appointed in writing 
and is required to complete training requirements. For all of the Project Maven 
contracts, the KO appointed the AWCFT Chief to serve as the TM. The TM works 
closely with the COR and oversees the day-to-day execution of the contracts and 
regularly communicates and coordinates with the COR for all of the contracts. 
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Finding 


(U) Finding 


(U) The Algorithmic Warfare Cross-Functional 
Team Successfully Monitored and Managed 
Project Maven Contracts, but Lacked Documented, 
Formalized Processes 


(U) The ACC-Aberdeen Proving Ground Contracting Center KO appointed a COR 

for Project Maven’s four contracts and one cooperative agreement in accordance 
with the FAR, and the ARL COR and the AWCFT TM successfully monitored 

and managed Project Maven’s four contracts and one cooperative agreement in 
accordance with the FAR, DFARS, DGARs, and contract requirements. In addition, 
the COR and TM successfully monitored and managed the contracts and agreement 
to ensure compliance with the PWS and QASP and safeguarded the interest of the 
Government’s contractual relationship by monitoring contract deliverables. 


(U) In addition, the TM, with the support of the AWCFT, actively monitored 
contract deliverables using AWCFT-developed reporting, metrics, processes, and 
procedures to meet Project Maven objectives in a fast-moving AI development, 
testing, and implementation environment. For example, the AWCFT engaged in 
regular project performance reviews and conducted frequent and transparent 
programmatic reviews to highlight performance and status, including financial and 
budgetary elements, with the ESG and DoD stakeholders. Furthermore, the AWCFT 
continuously engaged with contractors and regularly inspected the algorithms’ 
performance for each iteration of fielding and testing. Additionally, the TM and 
AWCFT financial manager conducted a detailed review of all contract financial 
reporting on a weekly basis. 


(U) However, the AWCFT did not document its approach to contract oversight 

by formalizing, in writing, the reporting metrics and procedures for monitoring 
and managing Project Maven contracts. The AWCFT used provisional reporting 
metrics, processes, and procedures because AI and machine learning is an 
emerging, complex, and rapidly moving technology that requires close monitoring 
and management techniques that are not captured in current procedures and best 
practices that are used by the DoD acquisitions community. 


(U) Without formalized and documented processes, there is an increased risk of 
lapses occurring in the monitoring and management of the Project Maven contracts 
as the program grows and as personnel change. This could negatively affect the 
long-term success and growth of the project. In addition, as Al and machine 


Cut 
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Finding 


(U) learning programs expand within the DoD, future DoD acquisitions related 
to this complex, rapidly-moving technology may not benefit from the AWCFT’s 
monitoring and management lessons learned. Best practices that can further 
refine and develop these types of acquisitions, as well as help provide continuity 
for management and monitoring of emergent technology, could be lost if these 


processes are not documented. 


(U) The Algorithmic Warfare Cross-Functional Team 
Managed Project Maven Contracts in Accordance With 
the FAR, DFARS, DGARs, and Contract Requirements 


(U) The ACC-Aberdeen Proving Ground Contracting Center KO appointed a COR 

for Project Maven’s four contracts and one cooperative agreement in accordance 
with the FAR, and the ARL COR and the AWCFT TM successfully monitored 

and managed Project Maven’s four contracts and one cooperative agreement in 
accordance with the FAR, DFARS, DGARs, and contract requirements. In addition, 
the COR and TM successfully monitored and managed the contracts and agreement 
to ensure compliance with the PWS and QASP and safeguarded the interest of the 
Government’s contractual relationship by monitoring contract deliverables. 


(U) The Contracting Officer Incorporated a PWS and a QASP 
With Reporting Requirements 

(U) The contractors provided PWSs for the four contracts, which the KO 
incorporated into the contracts upon award. The PWSs incorporated several 
reporting requirements and checkpoints, and the QASPs reflected the PWSs. 


E 


The developers and the integration team repeated this rapid iteration process 
every 6 months, which yielded an enhanced algorithm with each iteration. 


PY The TM and AWCFT conducted the inspections through 


i 
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{E84 the review of testing and integration reports and algorithm modeling results. 
TM inspection reports were provided to the COR who maintains a quality assurance 
file as required by the QASP. 


| 


The TM and AWCFT personnel 
conducted inspections through participation in program and technical reviews 


and the review of algorithm model plans. TM reporting was accomplished 
through meeting minutes and model plans that were provided to the COR, who 
maintained a quality assurance file as required by the QASP. In addition, the QASP 
provided details on the required service; performance standards, as stated in the 
PWS; acceptable quality levels; and method of surveillance for the performance 
requirements summary. 


(U) The COR Monitored the Contracts With the Support of the 
Appointed TM 


£684 In addition, weekly technical meetings were conducted with the COR, TM, 
and contractor to review the contractor’s technical progress, planned expenditures, 


and overall contract financial status. 


The COR and TM stated 
that they established a weekly meeting with the contractor that thoroughly covered 


all aspects of the contract. The COR, TM, and contractor met weekly because of 
the novel process of AI development, the need to closely monitor the contract as it 
was progressing, and the QASP requirements for the acceptance and surveillance 
standards. During these weekly meetings, the COR, TM, and contractor also 
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{EU discussed and identified contractor progress and performance. P 


= = The COR coordinated with the TM to review the 


deliverables to ensure requirements were met in accordance with the QASP and 


whether the contractor met performance standards. Additionally, during the 
weekly meetings with the contractor, both the COR and TM provided detailed 
customer feedback and discussed any performance issue in which deliverables did 
not meet contract requirements. 


(U) The Appointed TM Directly Monitored the Maven Contracts 
(U) The TM conducted regular, in-depth meetings with the contractor project 
managers to review projected deliverables, financials, and potential project risks. 
As outlined in the QASP, the TM is responsible for verifying that the contractor 
performs the technical requirements of the contract in accordance with the 
contract terms and conditions and surveilling the contractor’s performance. 
During an interview, the COR stated that the TM spent a considerable amount 

of time monitoring contractor performance because the research and development 
required for AI development and machine learning is not widely understood 
across the DoD and requires a heavy degree of oversight and monitoring. The COR 
attributed this level of effort and oversight of the contracts by the TM to the rapid 
pace and technological novelty of Al development. For example, in interviews 
with the COR and the TM, both acknowledged that the technology behind Project 
Maven’s mission objectives is new to the DoD and that the AWCFT was learning 
how to work at the speed of commercial development within the DoD enterprise. 


(U) The AWCFT Actively Monitored Project Maven 
Contract Deliverables, but the AWCFT Monitoring 
Procedures Were Provisional and Not 

Formally Documented 

f{€U8 The TM, with the support of the AWCFT, actively monitored contract 


dcliverab es, 
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(U) AWCFT Developed Provisional Metrics, Processes, and 
Procedures to Maintain the Pace of Al Development 

{6848 The AWCFT used provisional reporting metrics, processes, and procedures 
because AI and machine learning is an emerging, complex, rapidly-moving 
technology that requires close monitoring and management techniques that 

are not captured in current procedures and best practices that are used by the 


DoD acquisitions community. 


. The AWCFT also engaged 


with the contractor on a daily basis, conducted extensive weekly meetings, and 


reviewed each research line in detail to monitor progress, discuss and solve 
project obstacles, and coordinate on how best to deliver research capabilities 


to the customer. 


E 
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Project Maven O-6 Working Group, the ESG, and DoD stakeholders. 


{EG AWCFT DTM 18-00 


2 


13 (U) In software and technology development, a “sprint” refers to a develop iteration of specified duration, wherein 
a development team works to complete specific tasks, milestones, or deliverables. For Project Maven each sprint is 
90 days. 
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Cul Finding 


{EU AWCFT personnel agreed with the necessity of formalizing the procedures 
and shared draft plans to facilitate the transition of Project Maven to a mission 
owner with the evaluation team. The AWCFT Chief provided the evaluation team 
with information on the actions taken based on the discussions with the evaluation 
team and the finalization of the transition to the mission owner. See AWCFT 
Actions Taken for Recommendation 1. 


(U) Without Formalized and Documented Processes, There 

Is Increased Risk of Lapses Occurring in the Monitoring and 
Management of Project Maven 

(U) Without formalized and documented processes, such as standard operating 
procedures for monitoring and managing AI and machine learning contracts, 
there is an increased risk of lapses occurring in the monitoring and management 
of the Project Maven contracts as the program grows and personnel change. 
These lapses could negatively affect the long-term success and growth of the 
project. In addition, as AI and machine learning programs expand within the DoD, 
future DoD acquisitions relating to this complex, rapidly moving technology may 
not benefit from the monitoring and management lessons learned. Best practices 
that can further refine and develop these types of acquisitions, as well as help 
provide continuity for emergent technology monitoring and management, could be 
lost if they are not documented. 


{€UH The AWCFT has captured some lessons learned in its effort to build and 


train algorithms to meet mission objectives. aaa eee 


However, the lessons learned on 
Project Maven so far remain within the AWCFT. As the DoD embraces AI and 
accelerates the adoption of data management, analytics, and machine learning 


capabilities, the lessons learned from Project Maven may prove valuable to 
ensuring effective and responsible AI development across the Department. 


DODIG-2022-049 | 13 


Finding 


(U) Recommendations, Management Comments, and 
Our Response 


(U) Recommendation 1 

(U) We recommend that the Chief of the Algorithmic Warfare Cross-Functional 
Team formalize Project Maven’s processes and procedures for monitoring and 
managing artificial intelligence development contracts to ensure knowledge 
management, continuity, and efficiency when the project transfers to a future 
mission owner and for reference by subsequent novel technology projects 

and cross-functional teams. These formalized processes and procedures 
should capture: 


a. (U) the strategy and approach for managing, surveilling, and 
monitoring such projects; 


b. (U) contract monitoring procedures that surpass normal 
acquisition requirements; 


c. (U) conducting financial reviews; 
d. (U) partnering with and managing contractors; 
e. (U) reporting requirements; and 


f. (U) surveillance needs, including manpower and training 
requirements, to effectively manage, monitor, and 
surveil Project Maven. 


(U) Algorithmic Warfare Cross-Functional Team Actions Taken 
{€UH Throughout the evaluation, we discussed the need to formalize Project 
Maven’s processes and procedures with AWCFT personnel and the AWCFT shared 
draft documentation in support of that need. On October 14, 2021, we discussed 
the specific recommendations of this report with the AWCFT. During the 
discussion, the AWCFT Chief stated that the Joint Requirements Oversight Council 


Memorandum supporting the transition of Maven’s —E a 
ha = = | Directly following that discussion, the AWCFT provided 


evidence of actions taken to address the recommendation in this report, including 


a description of roles and responsibilities, standard operating procedure, and a 


Maven acquisition guide. In addition, it provided Bate 


BE These actions meet the intent of the recommendation; therefore, the 
recommendation is closed. 
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(U) Recommendation 2 

(U) We recommend that the Assistant Secretary of Defense for Acquisition 
conduct a review of the Algorithmic Warfare Cross-Functional Team’s 
emerging technology acquisition processes and procedures to evaluate 
whether any of the processes and procedures used by the Algorithmic 
Warfare Cross-Functional Team should be further formalized in acquisition 
policies, implemented by other programs and cross-functional teams, 

or incorporated into relevant curriculum and training. 


(U) Management Comments 

(U) The Assistant Secretary of Defense for Acquisition agreed with the 
recommendation and outlined several potential best practices for possible inclusion 
in the Software Acquisition Pathway in DoD Instruction 5000.87. To support this 
effort, the Office of the Assistant Secretary of Defense for Acquisition will interact 
with the Project Maven team to gather additional details, such as approaches that 
dynamically handle changing performance metrics and needs, and determine if and 
how those approaches can be elaborated upon in DoD acquisition guidance. 


(U) Our Response 

(U) The comments from the Assistant Secretary of Defense for Acquisition 
addressed the specifics of the recommendation. Therefore, the recommendation 

is resolved, but will remain open. We will close the recommendation when the 
Assistant Secretary of Defense for Acquisition provides us the results of the review 
of Project Maven’s practices and resulting determination of whether those practices 
should be incorporated into DoD acquisition guidance. 
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(U) Appendix 
(U) Scope and Methodology 


(U) We conducted this evaluation from October 2020 through October 2021 

in accordance with the “Quality Standards for Inspection and Evaluation,” 
published in January 2012 by the Council of Inspectors General on Integrity and 
Efficiency. Those standards require that we adequately plan the evaluation to 
ensure that objectives are met and that we perform the evaluation to obtain 
sufficient, competent, and relevant evidence to support the findings, conclusions, 
and recommendations. We believe that the evidence obtained was sufficient, 
competent, and relevant to lead a reasonable person to sustain the findings, 
conclusions, and recommendations. 


(U) This report was reviewed by the DoD Component associated with this oversight 
project to identify whether any of their reported information, including legacy 
FOUO information, should be safeguarded and marked in accordance with the DoD 
CUI Program. In preparing and marking this report, we considered any comments 
submitted by the DoD Component about the CUI treatment of their information. 

If the DoD Component failed to provide any or sufficient comments about the CUI 
treatment of their information, we marked the report based on our assessment of 
the available information. 


(U) Interviews with Officials 

(U) We interviewed individuals from the following organizations to determine: 
1) whether the ACC, Aberdeen Proving Ground Contracting Center KO; the 

ARL COR; and the AWCFT TM successfully monitored and managed the Project 
Maven four contracts and one cooperative agreement in accordance with the 
FAR, DFARS, DGARs, and contract requirements; 2) whether the KO, COR, and TM 
monitored and managed the contracts in compliance with the PWS, QASP; and 

3) whether the KO, COR, and TM safeguarded the interest of the Government’s 
contractual relationship by monitoring contract deliverables. 


e (U) Army Contracting Command - Aberdeen Proving Ground, Maryland 
e (U) Army Research Laboratory - Adelphi, Maryland 
e (U) Algorithmic Warfare Cross-Functional Team - Washington, D.C. 
(U) Due to COVID-19 travel restrictions, we did not physically visit any of the 
DoD Components identified in this report. We relied upon teleconferences, 


interviews, and data calls to collect testimonial and documentary evidence 
to verify information. 
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(U) Contract Selection and Documentation Review 

(U) We identified the contracts selected for this evaluation through a 
self-initiated followup on the evaluation Report No. DODIG-2020-025, 
“Evaluation of the Algorithmic Warfare Cross-Functional Team (Project Maven),” 
issued November 8, 2019. 


(U) To determine whether the KO, COR, and TM successfully monitored and 
managed the four contracts and one cooperative agreement, we requested and 
obtained the following documents: 


e (U) PWS for the KUBERA, BRONZE, PAVEMENT, and MORSE contracts; 
e (U) QASP for the KUBERA, BRONZE, PAVEMENT, and MORSE contracts; 
e (U) Research Plan for the CMU cooperative agreement; 

e (U) weekly, monthly, and quarterly reports for each contract; 

e (U) financial reporting or invoices for each contract; 

e (U) COR and TM appointment letters; 

e (U) narrative statements provided by the CORs; and 

e (U) AWCFT presentations to the O-6 Working group and ESG. 


(U) We reviewed the documents in comparison with the requirements outlined in 
the FAR, DFARS, the DGARS, PWS, and QASP. 


(U) Use of Computer-Processed Data 


(U) We did not use computer-processed data to perform this evaluation. 


(U) Prior Coverage 


(U) During the last 5 years, the DoD Office of Inspector General (DoD OIG) 
issued three reports discussing AI, information technology, or Project Maven 
related topics. Unrestricted DoD OIG reports can be accessed http://www.dodig. 


mil/reports.html/. 


(U) DoD OIG 
(U) Report No. DODIG-2020-098, “Audit of Governance and Protection of 
Department of Defense Artificial Intelligence Data and Technology” June 29, 2020 


(U) The DoD OIG determined that the Joint Artificial Intelligence Center (JAIC) 
had taken some steps to develop an AI governance framework and standards 
and offered six recommendations to ensure that the JAIC could meet the 
responsibilities outlined in the FY 2019 National Defense Authorization Act. 
Furthermore, the DoD OIG identified that the four DoD Components and 


Cut 


Appendix 
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(U) two contractors did not consistently implement security controls to protect 
the data used to support AI projects and technologies from internal and 


external cyber threats. 


(U) Report No. DODIG-2020-025, “Evaluation of the Algorithmic Warfare 
Cross-Functional Team (Project Maven)” November 8, 2019 


(U) This report is classified. To file a Freedom of Information Act 
request, please submit a request to FOIA Online. This report is a result of 
Project No. D2019-DISPAZ-0032. 


(U) Report No. DODIG-2018-110, “Defense Contract Management Agency’s 
Information Technology Service Contracts” April 25, 2018 


(U) The DoD OIG determined that Defense Contract Management 

Agency (DCMA) contracting officials did not properly award 11 of 14 
Information Technology (IT) service contracts valued at $61 million. 
Moreover, DCMA officials used flexible ordering agreements to award 

5 of the 14 contracts, which violated relevant FAR requirements. As a result, 
DCMA contracting officials acquired $56.4 million in IT services on contracts 


with poorly defined or nonexistent performance work statements. 
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(U) Management Comments 


(U) Assistant Secretary of Defense for Acquisition 


OFFICE OF THE UNDER SECRETARY OF DEFENSE 


3000 DEFENSE PENTAGON 
WASHINGTON, DC 20301-3000 


ACQUISITION 24-Nov-2021 


AND SUSTAINMENT 
MEMORANDUM FOR DEPARTMENT OF DEFENSE INSPECTOR GENERAL 


SUBJECT: Response to DoDIG Draft Report on Evaluation of Contract Monitoring and 
Management for Project Maven (Project No. D2021-DEVOPD-0023.000) 


As requested, I am providing responses to the general content and recommendations 
contained in the subject report. 


Recommendation 1: 

We recommend that the Chief of the AWCFT formalize Project Maven’s processes and 
procedures for monitoring and managing AI development contracts to ensure knowledge 
management, continuity, and efficiency when the project is transferred to a mission owner and 
for reference by subsequent novel technology projects and cross-functional teams. 


Recommendation 2: 

We recommend that the Assistant Secretary of Defense for Acquisition conduct a review of the 
Algorithmic Warfare Cross-Functional Team’s emerging technology acquisition processes and 
procedures. The review should evaluate whether any of the processes and procedures used by the 
Algorithmic Warfare Cross-Functional Team should be further formalized in acquisition 
policies, implemented by other programs and cross-functional teams, or incorporated into 
relevant curriculum and training. 


Response: Concur with comment. I do not believe this response is exempt from release. 


Our response is based on the information provided in the report. A deeper understanding of the 
degree to which the detailed AWCFT processes and metrics should be incorporated into DoD- 
wide policy or guidance would require further interaction with the project team. 


The report finds that Project Maven successfully monitored and managed its four contracts and 
one cooperative agreement in accordance with the applicable FAR, DFARS, DGARS, and 
contract requirements. It noted that the team actively monitored contract deliverables using 
reporting, metrics, processes, and procedures to meet Project Maven objectives in a fast-moving 
AI development, testing, and implementation environment. These approaches could provide 
valuable lessons learned and help establish best practices for other DoD Artificial Intelligence 
(AD)/Machine Learning (ML) efforts. That said, we also recognize that DoD is not limited to 
FAR-based contracting, and that there may be a variety of contracting authorities and associated 
monitoring approaches that DoD programs may consider. Our supporting key findings follow. 


The draft report discusses aligning reporting artifacts with contracts/Quality Assurance 
Surveillance Plans (QASPs). In keeping with Agile software development practices, programs 
should be afforded latitude for dynamic changes as a program evolves and learns. The QASP 
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(U) Assistant Secretary of Defense for Acquisition (cont’d) 


could be leveraged as an umbrella artifact, with specific performance evaluation processes and 
criteria established as part of planning sessions at the start of each sprint/iteration (much like the 
agile software development concept of “definition of done” is mutually agreed upon by 
government/contractor for each development activity). 


Additionally, the DoDI 5000.87 Software Acquisition Pathway demonstrates how such issues 
can be documented in a dynamic way that supports accountability and speed of delivery at the 
same time. Just as requirements are expected to evolve in an Agile development environment, so 
should the corresponding assessment criteria, especially in the fast moving AI space. Maven has 
made great progress in identifying applicable assessment criteria; documenting the evaluation 
criteria and method of using the QASP as an overarching mechanism as best practices for 
specific AI development activities will be valuable to future DoD programs. 


The Software Acquisition Pathway in DoDI 5000.87 is purpose built for innovation and speed in 
the digital age, and that policy and its associated guidance already generally aligns with many of 
the report’s best practices (e.g., tight iterations, frequent reviews, empowered cross-functional 
teams, dynamic requirements and specific valuation criteria). As such, A&S can interact with 
the Project Maven team to gather further details concerning the report’s best practices and then 
determine if and how they can be elaborated upon in A&S guidance. Such potential best 
practices for programs that A&S can explore further with Project Maven (and that are generally 
tenets of the Software Acquisition Pathway) include: 
1. Adapt but Update: explore dynamic use of different metrics, processes and procedures for 
ensuring a quality AI/ ML product is delivered to the Government. 
2. Frequent and Open Reviews: conduct frequent and transparent project performance and 
programmatic reviews to highlight performance and status. 
3. Engage and Inspect: engage with contractors and inspect an algorithms’ performance for 
each iteration of fielding/testing. 
4. Cross-Functional Teams Emphasize Speed: integrate roles into a small empowered team 
focused on continuous improvement, to deliver better and faster solutions. 
5. Identify Expertise: document desired skills and expertise in selecting a Contracting 
Officer’s Representative (COR) or Technical Monitor (TM). For a TM, the program 
should specify what areas of AI they should be monitoring based on their expertise. 


The Office of the Assistant Secretary of Defense for Acquisition has completed review of the 
DoD Inspector General’s draft report on Evaluation of Contract Monitoring and Management for 
Project Maven and concurs with the findings and recommendations in the report. 


Please EE ona 


information is required. 


Sincerely, 


ODONNELL.CHRI Distaly signed by 


DONNELL.CHRISTOPHER. 


STOPHER.C] c 


Date: 2021.11.24 10:36:57 
-05'00' 


Christopher C. O'Donnell 
Performing the Duties of Assistant Secretary 
of Defense for Acquisition 
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(U) Acronyms and Abbreviations 


ACC 
Al 

ARL 
AWCFT 
COR 
DFARS 
DTM 
ESG 
FAR 
KO 
PWS 
QASP 
™ 


Army Contracting Command 

Artificial Intelligence 

Army Research Laboratory 

Algorithmic Warfare Cross-Functional Team 
Contracting Officer’s Representative 
Defense Federal Acquisition Regulation Supplement 
Directive-type Memorandum 

Executive Steering Group 

Federal Acquisition Regulation 

Contracting Officer 

Performance Work Statement 

Quality Assurance Surveillance Plan 


Technical Monitor 


Acronyms and Abbreviations 
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Whistleblower Protection 
U.S. DEPARTMENT OF DEFENSE 


Whistleblower Protection safeguards DoD employees against 
retaliation for protected disclosures that expose possible fraud, waste, 
and abuse in Government programs. For more information, please visit 
the Whistleblower webpage at http://www.dodig.mil/Components/ 
Administrative-Investigations/Whistleblower-Reprisal-Investigations/ 
Whisteblower-Reprisal/ or contact the Whistleblower Protection 


Coordinator at Whistleblowerprotectioncoordinator@dodig.mil 


For more information about DoD OIG 
reports or activities, please contact us: 


Congressional Liaison 
703.604.8324 


Media Contact 
public.affairs@dodig.mil; 703.604.8324 


DoD OIG Mailing Lists 
www.dodig.mil/Mailing-Lists/ 


Twitter 
www.twitter.com/DoD_IG 


DoD Hotline 
www.dodig.mil/hotline 
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